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DETAILED ACTION 



1 . This action is responsive to communication: amendment filed 3/22/04 and the 
declaration filed 3/31/04 to the application filed on 05/30/97. 

2. Claims 101-109 are added. 

3. Claims 1-11, 13-16, 31-50, 63-66, 100-109 are pending in the case. Claims 1,10 
and 100 are independent claims. 

4. The rejections of claims 1-4, 9-11, 13-16, 63-66, 100 under 35 U.S.C. 103(a) as 
being unpatentable over Robotham have been withdrawn in view of the declaration. 

5. The rejections of claims 5-6 under 35 U.S.C. 103(a) as being unpatentable over 
Robotham and further in view of Robotham have been withdrawn in view of the 
declaration. 

6. The rejections of claims 7-8 under 35 U.S.C. 103(a) as being unpatentable over 
Robotham and further in view of Ando have been withdrawn in view of the declaration. 

7. The rejections of claims 31-50 under 35 U.S.C. 103(a) as being unpatentable 
over Robotham and further in view of Caire have been withdrawn in view of the 
declaration. 



8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 



Claim Rejections - 35 USC § 103 
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9. Claims 1-3, 7-1 1 , 14, 63-66, 100-109 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Near et al. (US Pat No. 5,995,091 , 1 1/30/99, filed 5/10/96). 
Regarding independent claim 1 , Near discloses: 

- receiving a stream including a file that integrates media content with 
choreography information within each of at least two objects of the file, each of 
the objects including media content data and choreography information 
associated therewith, the choreography information comprising data indicating an 
author-designated relationship between the objects of the file that defines an 
author-designated temporal order of presentation between the objects (figure 2, 
col 3, lines 28-47, col 9, lines 51-67: the interleaved playback data stream 
includes a plurality of media portions such as video chunk 1 , audio chunk 1 , 
video chunk 2, video chunk 3 where each portion has associated timestamp, 
engine destination, and message; each portion also has a simulation pointer for 
linking and arranging media portions in the stream, this shows the choreography 
information associated with media portion and the temporal order of display 
among the objects; the playback data stream indicates that the stream is 
received for playback; the authoring tool used by an author for specifying images 
and sounds for playback at specified times indicates an author-designated 
temporal order of presentation of the multimedia elements) 

- before all objects of the file are received, beginning to render media content 
encapsulated within the file based on the choreography information associated 
with objects received so as to enable display of the objects received based on 
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the temporal order defined by the choreography information, wherein the 
temporal order is maintained independent of a recipient or a web server (col 3, 
line 64 to col 4, line 27, col 7, lines 50-64: the fact that the playback of the 
images and sound data stream with specified times is generated by an author for 
a selected bandwidth shows that the temporal order is defined by the author, and 
thus is maintained independent of a recipient) 
Near does not explicitly disclose that the temporal order is maintained independent of a 
bandwidth of a communication channel used to send the multimedia document. 
However, it would have been obvious to one of ordinary skill in the art at the time of the 
invention was made to have modified Near to include the independence of the temporal 
order over a bandwidth of a communication channel used to send the multimedia 
document for the following reason. The fact that Near provides generating a bandwidth- 
controlled presentation data stream, thereby controlling network load and providing 
predictable performance on a variety of playback systems (col 2, lines 61-67) suggests 
that the playback of the data stream in Near is not dependent on the playback systems 
and is not dependent on a bandwidth of a communication channel used to send the 
multimedia document since the network load while sending media data is controlled via 
the authoring tool. 

Regarding claims 2 and 3, which is dependent on claim 1 , Near discloses: 

- changing one object in the data file (col 10, lines 1-27: update the multimedia 
elements in the playback data stream) 
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- adding an object to the data file (col 10, lines 1-27: "after processing the updated 
multimedia elements into portions ... the existing stream portions is replaced with 
the newly updated stream portions...",) 

Regarding claim 7, which is dependent on claim 1, Near discloses that each object has 
an address indicating a player that plays the object (col 8, lines 1-13; col 4, lines 47- 
61 identifying images and sounds to be reproduced during the playback implies 
identifying the address of the media element and the software that plays the media). 

Regarding claim 8, which is dependent on claim 1 , Near discloses compressing 
information in each object (col 8, lines 13-29, col 9, lines 20-39; col 4, lines 47-61). 

Regarding claim 9, which is dependent on claim 1, Near discloses creating an object in 
the file (col 9, lines 20-50) and locating player data within an object defining a player 
that plays the object (col 10, lines 40-67: playback media software used to identify 
individual portion of the playback data stream and the manager software calls a function 
for a player element of media data). 

Near does not disclose the created object is an unknown object. However, it would 
have been obvious to one of ordinary skill in the art at the time of the invention was 
made to have modified Near to include the created object being an unknown object for 
the following reason. Near defines the multimedia data for the playback data stream 
where each multimedia element is considered as an object in the data stream (figure 2, 
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col 51-67). Thus no matter the object is known or unknown, the system always locates 
the player associated with the multimedia element. 

Independent claim 10 is for a computer system of the method claim 1 , and is rejected 
under the same rationale. 

Regarding claim 1 1, which is dependent on claim 10, Near discloses that at least one 
object comprises one of a textual file format, an image file format, and a sound file 
format (figure 2, col 9, lines 50-67: at least one media element comprises one of audio 
format). 

Regarding claim 14, which is dependent on claim 10, Near discloses that each object is 
a generic element of the hierarchical data file structure, such that any combination of 
objects can be grouped together to form a part of the multimedia document (figure 2, col 
9, lines 50-67: each multimedia element is a generic element such as video element or 
audio element and the combination of a video element and an audio element can be 
grouped together under a topic to form a part of the multimedia document). 

Regarding claims 63 and 65, which are dependent on claims 1 and 10 respectively, 
Near discloses that the ordered display is independent of a recipient software program 
used to render the objects (col 3, line 64 to col 4, line 27, col 7, lines 50-64: the 
playback of the images and sound data stream with specified times is generated by an 
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author for a selected bandwidth, and thus the temporal order defined by the author, is 
maintained independent of a recipient software). 

Regarding claims 64 and 66, which are dependent on claims 63 and 65 respectively, as 
mentioned in claims 63 and 65 above, the ordered display is independent of the 
browser used to render the objects so that the display is presented as defined by the 
document author. 

Regarding independent claim 100, Near discloses: 

- generating a single file that integrates the media content with the choreography 
information, wherein generating the single file comprises encapsulating within the 
single file at least two objects, each object including media content data and 
choreography information associated therewith, the choreography information 
comprising data defining a temporal order of presentation between the objects 
(figure 2, col 3, lines 28-47, col 9, lines 51-67: the interleaved playback data 
stream includes a plurality of portions such as video chunk 1 , audio chunk 1 , 
video chunk 2, video chunk 3 where each portion of media has associated 
timestamp, engine destination, message, and simulation pointer for linking and 
arranging media portions in the stream shows the choreography information 
associated with media portion and the temporal order of display between the 
objects, and where the playback data stream indicates that the stream is 
received for playback; the authoring tool used by an author for specifying images 
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and sounds for playback at specified times indicates an author-designated 
temporal order of presentation between media objects) 

- before all objects of the files are received by a user, enabling the user to begin 
rendering the media content encapsulated within the file according to the 
choreography information associated with objects received, wherein the temporal 
order is maintained independent of a recipient or a web server (col 3, line 64 to 
col 4, line 27, col 7, lines 50-64: the playback of the images and sound data 
stream with specified times is generated by an author for a selected bandwidth, 
and thus the temporal order defined by the author, is maintained independent of 
a recipient) 

Near does not explicitly disclose: 

- receiving specification of media content by a user 

- receiving designation by the user of choreography information that indicates at 
least an intended order of presentation for the specified media content 

Instead, Near provides the media content of the playback media stream, the order of 
presentation of the playback media stream defined by an author where the pointer data 
in each multimedia element for arranging the order of playing of the media element in 
the stream indicates the choreography information of each element (figure 2, col 9, lines 
50-67). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have modified Near to include receiving specification content and 
receiving designation of choreography information for the following reason. The fact 
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that the order of presentation of the media playback is defined by an author shows the 
designation of the choreography information and the presentation order of the playback. 
Further, defining the media playback suggests including a description of the media 
contents of the playback, which is equivalent to the specification of the media content of 
the playback. And it was obvious that the description as well as the order of 
presentation are received at the destination where the playback is performed. 

Regarding claim 101, which is dependent on claim 100, Near discloses that the 
temporal order is independent of a recipient input (col 3, line 64 to col 4, line 27, col 7, 
lines 50-64: the fact that the playback of the images and sound data stream with 
specified times is generated by an author for a selected bandwidth shows that the 
temporal order is defined by the author, and thus is maintained independent of a 
recipient). 

Regarding claim 102, which is dependent on claim 100, Near discloses that the 
temporal order is independent of a recipient hardware configuration (col 3, lines 48-55: 
since the temporal order is defined by an author, the author controls the playback, and 
thus independent of the recipient hardware configuration). 

Regarding claim 103, which is dependent on claim 100, Near discloses that the 
temporal order is independent of a recipient software configuration (col 3, lines 48-55: 
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since the temporal order is defined by an author, the author controls the playback, and 
thus independent of the recipient software configuration). 

Regarding claim 104, which is dependent on claim 1 , Near discloses that the temporal 
order is independent of a recipient input (col 3, line 64 to col 4, line 27, col 7, lines 50- 
64: the fact that the playback of the images and sound data stream with specified times 
is generated by an author for a selected bandwidth shows that the temporal order is 
defined by the author, and thus is maintained independent of a recipient). 

Regarding claim 105, which is dependent on claim 1 , Near discloses that the temporal 
order is independent of a recipient hardware configuration (col 3, lines 48-55: since the 
temporal order is defined by an author, the author controls the playback, and thus 
independent of the recipient hardware configuration). 

Regarding claim 106, which is dependent on claim 1 , Near discloses that the temporal 
order is independent of a recipient software configuration (col 3, lines 48-55: since the 
temporal order is defined by an author, the author controls the playback, and thus 
independent of the recipient software configuration). 

Regarding claim 107, which is dependent on claim 10, Near discloses that the temporal 
order is independent of a recipient input (col 3, line 64 to col 4, line 27, col 7, lines 50- 
64: the fact that the playback of the images and sound data stream with specified times 
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is generated by an author for a selected bandwidth shows that the temporal order is 
defined by the author, and thus is maintained independent of a recipient). 

Regarding claim 108, which is dependent on claim 10, Near discloses that the temporal 
order is independent of a recipient hardware configuration (col 3, lines 48-55: since the 
temporal order is defined by an author, the author controls the playback, and thus 
independent of the recipient hardware configuration). 

Regarding claim 109, which is dependent on claim 10, Near discloses that the temporal 
order is independent of a recipient software configuration (col 3, lines 48-55: since the 
temporal order is defined by an author, the author controls the playback, and thus 
independent of the recipient software configuration). 

10. Claims 4, 13, 15, 16 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Near as applied to claims 1 and 10 above, and further in view of Shaw et al., 
Microsoft Office 6-/n-f , Que Corporation 1994, pages 379-380, 384-389, 396-402, 419- 
425, 492-496. 

Regarding claim 4, which is dependent on claim 1, Near does not disclose: 

- creating an exclusionary area within the window 

- locating an object within the exclusionary area, the object being selected from a 
group of objects including a framed image, a slide show, framed text, sound data, 
a separator, or a hyperlink 
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Shaw discloses: 

- creating an exclusionary area within the window (page 401 , figure 4.4) 

- locating an object within the exclusionary area, the object being selected from a group 
of objects including a framed image, a slide show, framed text, sound data, a separator, 
or a hyperlink (page 401 , figure 4.4: the data in the area within the window can be text 
and graphics). 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Shaw into Robotham since Shaw teaches the window 
features for locating objects selected from a group of specific objects providing the 
advantage to include in Robotham to enhance the use of objects from internet instead 
of merely video or audio from the media database- 
Regarding claim 13, which is dependent on claim 10, Near does not disclose that two or 
more objects have at least one common attribute, including at least one of a command 
for perception of the object, an ability to pass and receive a message, and an ability to 
supply and retrieve the data embodied in the object. 

Shaw discloses that two or more objects have at least one common attribute, including 
at least one of a command for perception of the object, an ability to pass and receive a 
message, and an ability to supply and retrieve the data embodied in the object (page 
495: since the display of the slides can be set in a temporal order by the document 
author, the slides as in the slide stream has the ability to pass and receive a message to 
automatically advance to the next slide to display the data in the slide). 
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It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Shaw into Near since Shaw teaches the ability of passing 
and receiving a message between the objects to supply and retrieve data embodied in 
the object providing the advantage of enhancing the relationship among the media 
objects choreographed in the image stream as in Near. 

Regarding claims 15 and 16, which are dependent on claim 10, Near does not disclose 
the document forms a code segment that receives image information, and wherein the 
image information is used to construct an image frame for a framed image that is part of 
the multimedia document. 

Shaw discloses that the document forms a code segment that receives image 
information, and wherein the image information is used to construct an image frame for 
a framed image that is part of the multimedia document (pages 400-401). 
It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Shaw into Near since Shaw a code segment to control the 
receiving of image information as well as the structure of the media stream providing the 
advantage of utilizing such control to edit the media stream in Near as desired. 

11. Claims 5-6 are rejected under 35 U.S.C. 103(a) as being unpatentable over Near 
as applied to claim 1 above, and further in view of Johnson (US Pat No. 5,892,847, 
4/6/99, filed 4/22/96). 
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Regarding claims 5 and 6, which are dependent on claims 1 and 5 respectively, Near 
does not disclose defining as well as locating the update splash image within the data 
file. 

Johnson discloses: 

- splash image data defining a splash image and locating the splash image data within 
the data file for displaying the splash image on the computer display (col 4, lines 30-50) 

- further updating the splash image to be displayed (col 4, lines 30-63) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Johnson into Near since Johnson teaches the process of 
displaying of a splash image, which is an element of a multimedia document providing 
the advantage of including a splash image into the image file to enhance the image 
characteristics in Near. 

12. Claims 31-50 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Near as applied to claims 1 and 10 above, and further in view of Caire et al. (US Pat No. 
5,663,962, 9/2/97, filed 9/15/95). 

Regarding claim 31, which is dependent on claim 1, and claims 32-34, which are 
dependent on claim 31, Near discloses a header, timestamps, and associated data for a 
data stream (col 5, line 64 to col 6, line 16). Near does not disclose that the 
choreography information further comprises an object archive for storing information 
about one or more objects, the object archive including information about the 
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relationship of the object file with the document, and a multiplex section including data 
for the objects in the document. 
Caire discloses: 

- a header (col 1 , lines 65 to col 2, lines 1-2, each packet in the overall stream 
includes a header) 

- an object archive for storing information about the plurality of object files, the 
object archive including information about the level of each object file with the 
hierarchy (col 1 , lines 65 to col 2, lines 1-2, each packet of the multimedia stream 
stores information; col 1 , lines 37-52, it is desired for instance to insert into the 
complete stream also some subtitles to be displayed during the presentation....) 

- a multiplex section including data for each of the object files of the document (col 
1 , lines 65 to col 2, lines 1-9, 45-59) 

- the object files in the multiplex section are each played by a player as the 
multiplex object file is received by a receiver (col 1 , lines 65 to col 2, lines 1-2) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Caire into Near since Caire provides the choreography 
and the multiplexing features for a multimedia presentation. The combination of Caire 
into Near would provide more detailed options in the relationship of the objects in a 
multimedia document to effectively controlling and changing the presentation of the 
objects. 
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Regarding claims 35, 36-39, which are dependent on claims 31 and 35 respectively, 
Near does not disclose an object number counter indicating the number of objects, a 
plurality of object descriptions, each object description describing a corresponding one 
of the objects, and a choreography group providing information about a first group of 
objects, a group object counter indicating the number objects in the choreography 
group, size and type data for each object, header data, data slices of the objects 
interleaved together, and placing one or more slice size data blocks before one or more 
of the interleaved data slices, each slice size data block corresponding to a data slice 
and providing a size of the corresponding data slice. 
Caire discloses: 

- an object number counter indicating the number of object files (col 2, lines 10-20) 

- a plurality of object descriptions, each object description describing a 
corresponding one of the object files (col 1 , lines 65 to col 2, lines 1-2, the 
header includes information of the type of a packet in the multimedia stream) 

- a choreography group providing information about a first group of object files (col 
1, lines 65 to col 2, lines 1-2, packets of different types are included in the overall 
stream as a sequence of intervals wherein the type of a packet is disclosed in the 
heading are considered as a choreography group providing information about the 
object files) 

- size and type data for each object file (col 1 , lines 65 to col 2, lines 1-2, data type 
of each packet in the multimedia stream) 

- header data (col 1 , lines 65 to col 2, lines 1-2, each packet includes a header) 
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- the data slices of the object files interleaved together (col 1 , lines 65 to col 2, 
lines 1-2, the overall stream is structured as a sequence of intervals called 
packets, each of which contains data of single type, indicated in a header of the 
packet itself; since data of different types are arranged in the sequence of 
intervals called packets, the packets which are equivalent to the object files, are 
interleaved together) 

- a first player pointer including an address of a player that plays the choreography 
group (col 2, lines 3-9, for each interval, the multiplexer has to decide from which 
the input stream it should take the data in order to construct the packets; this 
implies that the multiplexer has to decide where to point to play the overall 
stream which is equivalent to the choreography group as mentioned above) 

- locating a plurality of slice size data blocks before the interleaved data slices, 
each slice size data block corresponding to one of the data slices and providing a 
size of the corresponding data slice (col 4, lines 45-53, the number of data bytes 
and the number of header bytes in each packet show the size of each packet 
which is equivalent to the data block) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Caire into Near since Caire provides the choreography 
and the multiplexing features for a multimedia presentation. The combination of Caire 
and Near would provide the relationship of the objects in a multimedia document for 
effectively controlling and changing the presentation of the objects. 
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Regarding claim 40, which is dependent on claim 31 , Near does not disclose a non- 
multiplex section following the multiplex section where the non-multiplex section 
includes one or more separate objects that are not played by a player as the separate 
object files are received by a receiver. Caire discloses a plurality of separate object files 
that are not played by a player as the separate object files are received by a receiver 
(col 1 , lines 37-45, ..Video and audio information have to be separated again, by an 
inverse of demultiplexing process, as presentation occurs on different devices...). 
It would have been obvious to one of ordinary skill in the art at the time of the invention 
was made to have combined Caire into Near since Caire provides the choreography 
and the multiplexing features for a multimedia presentation. The combination of Caire 
and Near would provide the relationship of the objects in a multimedia document for 
effectively controlling and changing the presentation of the objects. 

Claims 41-50 are for a computer system of the method claims 31-40, and are rejected 
under the same rationale. 

Response to Arguments 

13. Applicant's arguments with respect to claims 1-11, 13-16, 31-50, 63-66, 100 have 
been considered but are moot in view of the new ground(s) of rejection. 

14. The declaration filed on 3/31/04 under 37 CFR 1 .131 is sufficient to overcome 
the Robotham reference. 
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The declaration under 37 C.F.R. 1.131 filed on 3/31/04 and the amendment filed on 
3/22/04 have changed the scope of the claimed invention, and thus overcome the date 
of the prior art Robotham used in the previous office action. 
Therefore, the Examiner has to make a new ground of the rejections based the new 
scope the claimed invention and the amendment using the Near reference as in the 
rejections above. 

Conclusion 

1 5. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

16. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Crites et al. (US Pat No. 6,097,380, 8/1/00, filed 6/24/96). 
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Quershi et al. (US Pat No. 6,084,582, 7/4/00, filed 7/2/97). 

Nimri et al. (US Pat No. 5,990,931, 11/23/99, filed 4/9/97, priority 4/10/96). 

Nakajima et al. (US Pat No. 5,394,548, 2/28/95, filed 10/9/92). 

Ossenbruggen et al., Music in time-based hypermedia, ACM 1994, pgs. 224-227. 

Ackermann, Direct manipulation of temporal structures in a multimedia application 

framework, ACM 1994, pgs. 51-58. 

1 7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Cong-Lac Huynh whose telephone number is 703-305- 
0432. The examiner can normally be reached on Mon-Fri (8:30-6:00). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Heather Herndon can be reached on 703-308-5186. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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